1. Field of the Invention
The present invention relates to video processing, and, in particular, to the registration of video sequences.
2. Description of the Related Art
Video registration refers to the process of identifying the (e.g., temporal, spatial, and/or histogram) correspondence between two video sequences, e.g., an original video sequence and a processed video sequence generated from the original video sequence.
For many applications, such as watermark detection and reference-based video quality measurement, a processed video sequence may need to be registered to the original sequence. For example, to detect watermarks embedded in pirated videos that are shot using a camcorder, the processed video may need to be registered to the original one displayed in the theater. Another area where video registration is typically needed is in reference-based video quality measurement. To ensure quality of service (QoS), it is often necessary to measure the quality degradation between the original video and the one received by a client. The received video is often a processed version of the original video. Therefore, to achieve a meaningful reference-based quality measurement, the received video is first registered with respect to the original video sequence.
Differences between a processed video and the original one often result from a combination of spatial misalignment, temporal misalignment, and histogram misalignment. Spatial misalignment is the result of spatial manipulation of a video sequence, such as warping, cropping, and resizing (e.g., capturing a movie with 2.35:1 aspect ratio using a camcorder with 4:3 aspect ratio). The main causes of temporal misalignment are (1) the change of temporal resolution, such as frame rate conversion (e.g., 3-2 pull down), and (2) the dropping and/or repeating of frames used by video compression algorithms (e.g., MPEG-4). The video capturing process also causes temporal misalignment, because the displaying and the capturing generally (1) are not synchronized and (2) operate at different frame rates. In addition, processed videos in general have different color histograms from the original videos. This is often the result of video processing, such as compression, filtering, or gamma changes. It can also be the result of white balance or automatic gain control (AGC) in camcorder capture.
Spatial, temporal, and histogram registration can be used to correct the three types of misalignment. Spatial registration and histogram registration have been studied by many researchers. However, few studies have been done on temporal registration. A temporal registration scheme for video quality measurement was proposed in Jiuhuai Lu, “Image analysis for video artifact estimation and measurement,” Proc. of SPIE Machine Vision Applications in Industrial Inspection, v. 4301, pp. 166-174, San Jose, Calif., January 2001, the teachings of which are incorporated herein by reference. This scheme can recover the global offset between two video sequences. The global offset is estimated by maximizing the normalized correlation between temporal activity signatures extracted from each sequence. Caspi and Irani use a direct search for recovering sequence-level temporal misalignment, such as fixed shift or fixed frame rate conversion. See Y. Caspi and M. Irani, “Alignment of non-overlapping sequences,” Proc. of IEEE Int'l Conf. on Computer Vision, Vancouver, BC, Canada, July 2001.